I wonder if it wouldn't be better to decalare each shared variable instead of just the namespace. While more text to write, I think it would make it more clear how it works. And since this will remain a rarely used functionality, self explaining code might be more importat, than small code.

<?xml version="1.0" encoding="utf-8"?>
<MixxxControllerPreset ...>
    <info>
        ...
    </info>
    <sharedData namspace="S4MK3">
         <variable name="globalShift" type=boolean>
         <variable name="deckNo" type=number>
    </sharedData>
    <controller id="...">
       ...
    </controller>
</MixxxControllerPreset>

I have added namespace restriction. Hopefully this is heading toward the right direction.

I think it would make it more clear how it works

While this provide more visibility of what data is available in certain namespace, this is likely to make the devx less efficient as the mapping are static file, which you usually don't use actively while developing new controller mappings.

Thank you @JoergAtGithub for this thorough writeup. I have a couple comments:

Case2: This is a very valid usecase and to be honest, not one I had thought about before. The issue here is even more general in that sense that in that scenario, it would even be difficult to match which screen belongs to which controller (at least with PortMidi, since it doesn't expose the required information, right?).

Case 3, 4 and 5 are not supposed to be handled by this PR. There should not be (enforced by code review) a global namespace. I fully agree that the current solution does not work for "open-ended" interfaces. Those need to be centrally coordinated.

So far, I am only aware of two use cases for shared data between mappings

I also have a another one to add. the Traktor Kontrol S8 has buttons on the sides of the screens which likely communicate via the HID endpoint, but in practice only change the state of the QML screens.

Usecase 5 is also interesting because its slightly ambigous in the sense that we should not assume that the user would always want these layers to be in sync. maybe they would like Deck 1/2 to be purely their "Stems deck" controlled by the S8 (having access to all the stems features) and deck 3/4 controlled by the turntables to have their turntable control (for accurate scratching). In that case, whatever mechanism manages the "focused layer" needs to be able to also be ignored. But this discussion should probably be continued in a proposal setting.

TLDR; we need to address case 2 somehow, case 3,4,5 are and should not be solved in/via this PR.

Once such an API is in in the field, we have to maintain it forever due to backward compatibility issues

This (and the following about maintaining an API) is only true for stable API. In this case, we could release this feature and mark this feature as unstable to unblock S4 Mk3 implementation (which as a reminder is meant to be officially supported by Mixxx). This way you can freely remove those methods once there is a replacement.

I assume your point is also valid for #11407

I must say I find it disrespectful to play that card after 2 months of back and forth review processes.

So far, I am only aware of two use cases for shared data between mappings:

I guess you are taking Traktor/Native Instrument capability as reference. I suggest you look into #12557 or this mod for Traktor.

TL;DR: The scope for mapping synchronisation is infinite.

My suggestion would be that we create a concept document at https://github.com/mixxxdj/proposals in which we draft an API design that covers all requirements.

That sounds reasonable, except that https://github.com/mixxxdj/proposals doesn't seem of great interest to the Mixxx community so far (see my stale PR trying to bring that repo to live). So till proven otherwise, I won't spend time draft a thorough proposal to see it being ignored.

This (and the following about maintaining an API) is only true for stable API. In this case, we could release this feature and mark this feature as unstable to unblock S4 Mk3 implementation (which as a reminder is meant to be officially supported by Mixxx). This way you can freely remove those methods once there is a replacement.

I agree, that it would be okay to mark this feature private. especially since its essentially just a workaround for the fact that we can't run multiple controllers in the same engine anyways. ergo, this should be replaced in the mid-term future anyways.

I must say I find it disrespectful to play that card after 2 months of back and forth review processes.

Let me assure you that @JoergAtGithub didn't mean for his veto to be disrespectful. His criticism is valid (especially considering the complications for case 2 which we should work through).

That sounds reasonable, except that https://github.com/mixxxdj/proposals doesn't seem of great interest to the Mixxx community so far (see mixxxdj/proposals#1 trying to bring that repo to live). So till proven otherwise, I won't spend time draft a thorough proposal to see it being ignored.

I'm sorry that the PR has gone stale for so long. It's just easy to loose track of things. @daschuer would you mind continuing your ongoing review in mixxxdj/proposals#1?

What is the status of this PR? Should I close it or is there something we can do to finish it?

I think case 2 is valid. Do you have an idea on how to identify two different controllers and match with screen belongs to which controller? I think once we have identified that, attaching that info (invisibly) to the namespace handles is straightforward. But we need some way of ensure that the screen of one device doesn't get matched to the midi ports of another device.

I'm sorry but no, it does not make sense, and quite frankly I've lost the motivation to try and explain why.
Since we refuse to work with iteration on that matter, I'm going to chose my battle on forfeit on this one.

For anyone reading this with a S4 Mk3 - note that I will keep my dev branch (traktor-s4-mk3) with this feature, as long as I keep using this device.

FYI @ywwg please note that it won't be possible to complete #13004 now and will need someone else to take on that feature.

I regret that this PR has been closed. As a reviewer, I also spent a lot of time on this topic.
About two months ago, I pointed out the problems with multiple devices of the same type, and suggested grouping different scripts via namespaces defined in the JavaScript domain as a possible solution.
This would have been a phased implementation approach, but would have allowed the API for sharing data to be defined now and the secondary APIs for defining the grouping criteria to be implemented later. This would have allowed different grouping criteria such as the USB serial number, or manual grouping of shared parameters in a future graphical user interface. Without discarding the newly introduced API for data sharing.
I have expressed my concerns openly without enforcing a particular solution, but the recent change to a single namespace hardcoded in the XML file, would have introduced an API which cannot be extended in the future. This would have been a public API, which we know from the start, we will have to replace in the foreseeable future. So I finally had to submit a formal request for change.

it does sound like a formal proposal is required for this to be resolved. Hopefully the ultimate solution is not so far off from this implementation that we can use this code.

@acolombier I appreciate your contributions and understand your frustration. With large, delicate new features such as the ones you are working on, it is often hard to do design in the PR process, which is why we need to make better use of the proposal feature. That way the implementation can be worked out before you have done a bunch of work.

In the past, when we have taken a step back from PR discussions and moved to a design discussion we have usually found that
1: people are not that far apart, and
2: the changes required to make everyone happy are not very much.

I am thinking of the hotcue color PR which was a huge issue back in the day. We ended up mapping out the two designs and almost immediately found a way through.

About two months ago, I pointed out the problems with multiple devices of the same type, and suggested grouping different scripts via namespaces defined in the JavaScript domain as a possible solution.

I'm sorry that I have previously missed your concerns in regard to the multiple devices issue.

This would have been a phased implementation approach, but would have allowed the API for sharing data to be defined now and the secondary APIs for defining the grouping criteria to be implemented later.

I disagree with your assessment that this was an option. Imposing any grouping criteria later would've limited the API again, resulting in a breaking change. Unless I'm misunderstanding your "grouping criteria" concept.

This would have allowed different grouping criteria such as the USB serial number, or manual grouping of shared parameters in a future graphical user interface.

We can still add the grouping criteria now, at least without breaking the API description. The namespace defined in the XML could just be part of a larger identifier that takes more info, such as the usb serial into account. That same extension would not be hard to retrofit into this PR imo.

I have expressed my concerns openly without enforcing a particular solution, but the recent change to a single namespace hardcoded in the XML file, would have introduced an API which cannot be extended in the future.

I disagree. Please elaborate.

Please elaborate.

If we would have the namespace as string in JavaScript like in

engine.getSharedData("GlobalNamspace", "Shift");

we could later add an API like

engine.getUsbSerialNo();

and could use it together with the unmodified sharing API like

engine.getSharedData("DeviceSpecificNamespace" + engine.getUsbSerialNo(), "DeckSwitch");

I see two fundamental issues with that:

1. It allows to specify a global namespace, something which as you rightfully pointed out doesn't work with the schema-less object being exchanged here. But I'm arguing that this PR is not supposed to address that usecase.
2. The controllerscripts manually need to take care of avoiding naming collisions. A mapping author might not think of that instead expecting mixxx to take care of that. It's reasonable to expect this collision handling to be taken account for controllers that are made to be used in tandem (single-deck controller like the Traktor D2 for instance), but it may not be reasonable for the S4 to take care of that (even though its totally possible to use two S4's with a single mixxx instance). Moreover, it may not be feasible to get the USB-serial number, so we'd need to enable some way of manual grouping from the UI as a fallback anyways IMO. This would not fit cleanly into your proposed design either.

"Viele Köche verderben den Brei"
This PR is really a can of worms, because everyone has a different idea.

Sorry, I was also involved in that here: #12199 (comment)
Our conclusion at that time was to keep this as a mapping private solution and consider a C++/Control Object API for other interoperable features that are usable for all mappings.

Thank you @JoergAtGithub to bringing all the cases up.
I have probably not fully understand everything, but for me they look like candidates for a global solution for all mappings. The user needs probably a GUI to set it up as well.

My hope is to strip that PR down to the original use case and just make sure we have not created a maintenance burden that will bite us in future as expressed.

Let him cook? What do you think.

"Viele Köche verderben den Brei" This PR is really a can of worms, because everyone has a different idea.

I had the same feeling. I appreciate we came to the same conclusion.

The user needs probably a GUI to set it up as well.

I think this makes sense. This way I can keep the namespace scoping I had already implemented following the general consensus on the need to isolate shared data, but keep it to a "default" for now.
In second phase, we could decide how we would want the mapping and GUI to look like regarding that need.

Per discussion, this work is not abandoned but on the back burner while we do a proper design for controller data sharing.